Papers with visual reasoning tasks
Beyond Visual Understanding Introducing PARROT-360V for Vision Language Model Benchmarking (2025.coling-industry)
Copied to clipboard
| Challenge: | Current benchmarks for evaluating Vision Language Models (VLMs) often fail to thoroughly assess these models’ abilities to understand complex visual and textual content. |
| Approach: | They propose a benchmark that features 2487 visual puzzles designed to test VLMs on complex visual reasoning tasks. |
| Outcome: | The PARROT-360V Benchmark features 2487 visual puzzles designed to test VLMs on complex visual reasoning tasks. |
ProtoVQA: An Adaptable Prototypical Framework for Explainable Fine-Grained Visual Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Visual Question Answering (VQA) is increasingly used in diverse applications where models must provide accurate answers and explanations that humans can easily understand and verify. |
| Approach: | They propose a unified prototypical framework that learns question-aware prototypes that serve as reasoning anchors and applies spatially constrained matching to ensure that the selected evidence is coherent and semantically relevant. |
| Outcome: | The proposed framework yields faithful, fine-grained explanations while maintaining competitive accuracy. |
From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating insightful explanations with limited annotations are limited. |
| Approach: | They propose a method that iteratively computes visual features, an answer, and an explanation to improve the explanation quality step by step until the answer converges. |
| Outcome: | The proposed method outperforms previous methods while utilizing 5% of the human-annotated explanations across 10 metrics, showing up to 4.2 and 1.3 increases in BLEU-1 score on the VCR and VQA-X datasets. |
Enhancing Advanced Visual Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Vision-Language (VL) research have sparked new benchmarks for complex visual reasoning, challenging models’ advanced reasoning ability. |
| Approach: | They propose a novel multi-modal in-context learning methodology to enhance LLMs’ contextual understanding and reasoning. |
| Outcome: | The proposed model achieves SOTA performance among all visual reasoning tasks and achieves a 'higher level of accuracy' than previous models. |
NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models have demonstrated their strong performance on IQ test questions, achieving high scores across many languages. |
| Approach: | They propose a dataset to evaluate cognitive multimodal reasoning and problem-solving skills of large models. |
| Outcome: | The proposed dataset contains 2,728 multiple-choice questions and 4,642 images spanning 26 categories. |
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that use natural language rationales provide intuitive, higher-level explanations that are easily understandable by humans. |
| Approach: | They propose a model that generates free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. |
| Outcome: | The proposed model generates free-text rationales by combining pretrained language models with object recognition, grounded visual semantic frames, and visual commonsense graphs. |
Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Current sycophancy research has largely overlooked its specific manifestations in the video-language domain. |
| Approach: | They propose a video-LLM sycophancy benchmarking and evaluation to evaluate scophancies in video-LLMs. |
| Outcome: | The proposed benchmark evaluates sycophantic behavior in state-of-the-art Video-LLMs across diverse question formats, prompt biases, and visual reasoning tasks. |
Beyond Embeddings: The Promise of Visual Table in Visual Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Visual representation learning has been a cornerstone in computer vision for decades. |
| Approach: | They propose a visual representation tailored for visual reasoning that provides instance-level world knowledge and detailed attributes that are essential for visual reason. |
| Outcome: | The proposed visual tables outperform existing models on 11 visual reasoning benchmarks. |
VIPHY: Probing “Visible” Physical Commonsense Knowledge (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that vision-language models can retain and generalize knowledge, but they do not measure their ability to retain it. |
| Approach: | They build an automatic pipeline to derive a knowledge resource for calibrating and probing vision-language models. |
| Outcome: | The proposed model outperforms the pretrained model on size and spatial tasks. |
Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that CLIP models struggle with visual reasoning tasks . despite the success of Contrastive Language-Image Pretraining, there are still limitations . |
| Approach: | They propose to use a visual encoder to train CLIP-like models for fine-grained visual reasoning tasks. |
| Outcome: | The proposed models outperform CLIP-like encoders in visual reasoning tasks . the study highlights the importance of VLM architectural choices . |
DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing jailbreak methods only use a single image, restricting the attack space . Existing frameworks only use single image to distribute harmful requests across multiple images . |
| Approach: | They propose a compositional jailbreak framework that leverages Distributed instruction, Multimodal evidence and a Number chain task to fully enhance the jailbreak performance. |
| Outcome: | The proposed framework achieves attack success rates of over 90% on GPT-4o, Gemini-2.5-pro and Claude Sonnet 4 . |
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning (2025.coling-main)
Copied to clipboard
Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, Ji-Rong Wen
| Challenge: | Experimental results show that visual instruction tuning improves performance of Multi-modal Large Language Models (MLLMs) to extend the application scope of Large Language Modells, a surge of work augments LLMs with vision encoders to endow the ability of multi-modal cognition and reasoning. |
| Approach: | They propose a systematic approach to create high-quality visual reasoning instructions using a synthesize-complicate-reformulate paradigm. |
| Outcome: | The proposed method improves performance of MLLMs by 27.86% and 27.60% on MME-Perception and MME Cognition. |
Curr-ReFT: Overcoming Training Bottlenecks in Small-scale Vision-Language Models via Curriculum Reinforcement Finetuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art vision-language models require massive scaling that limits practical deployment. |
| Approach: | They propose to use supervised fine-tuning to train small-scale vision-language models but face out-of-domain collapse when trained with traditional supervised learning (SFT). |
| Outcome: | Experiments show that curr-reFT achieves state-of-the-art performance across visual tasks in both in- and out-of domain settings and benchmarks. |
Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks (2024.acl-long)
Copied to clipboard
Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, Muhammad Abdul-Mageed
| Challenge: | MLLMs have proven effective in a wide range of tasks that require complex reasoning and linguistic comprehension, but they are limited to English-based settings. |
| Approach: | They propose a family of Arabic multimodal large language models with strong vision and language capabilities. |
| Outcome: | The proposed models show strong performance on visual reasoning tasks and language capabilities. |
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing scientific benchmarks lack human-annotated difficulty levels and structured taxonomies of scientific concepts. |
| Approach: | They propose a benchmark for evaluating mathematical and physical reasoning through text-only and text-image formats with human-annotated difficulty levels and detailed explanations. |
| Outcome: | The proposed model achieves only 63.77% accuracy and struggles with visual reasoning tasks. |
Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained large language models can be fine-tuned with instruction tuning to align the model responses with human intentions. |
| Approach: | They investigate the effectiveness of parameter efficient fine-tuning (PEFT) of the Q-Former with visual reasoning benchmarks ScienceQA and IconQA. |
| Outcome: | The proposed model achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. |
AdaTooler-V: Adaptive Tool-Use for Images and Videos (2026.findings-acl)
Copied to clipboard
Chaoyang Wang, Kaituo Feng, Dongyang Chen, Zhongyu Wang, Zhixun Li, Sicheng Gao, Meng Meng, Xu Zhou, Manyuan Zhang, Yuzhang Shang, Xiangyu Yue
| Challenge: | Existing models exhibit blind tool-use reasoning patterns, which significantly increases inference overhead and degrades model performance. |
| Approach: | They propose an MLLM that performs adaptive tool-use by determining whether a visual problem truly requires tools. |
| Outcome: | The proposed model outperforms existing methods in visual reasoning tasks. |
Counteracting the Matthew Effect in Self-Improvement of LVLMs through Head-Tail Re-balancing (2026.acl-long)
Copied to clipboard
Xin Guo, Zhiheng Xi, Yiwen Ding, Yitao Zhai, Xiaowei Shi, Xunliang Cai, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Large vision language models have impressive reasoning capabilities across complex multimodal tasks. |
| Approach: | They propose to use distribution-reshaping and trajectory-rebalancing to improve visual reasoning capabilities. |
| Outcome: | Experiments on Qwen2-VL-7B-Instruct and InternVL2.5-4B models show that their methods outperform baselines by 3.86 points. |
Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model (2024.emnlp-main)
Copied to clipboard
Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, Yueting Zhuang
| Challenge: | Using large language models, large multimodal models struggle with basic tasks like reading time from a clock and planning a route using a road map. |
| Approach: | They propose a multimodal self-instruct that synthesizes massive abstract images and visual reasoning instructions. |
| Outcome: | The proposed model synthesizes 11,193 abstract images and reasoning instructions across eight visual scenarios. |
Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts (2026.acl-long)
Copied to clipboard
Haolei Xu, Haiwen Hong, Hongxing Li, Rui Zhou, Yang Zhang, Longtao Huang, Hui Xue, Yongliang Shen, Weiming Lu, Yueting Zhuang
| Challenge: | Existing multimodal Mixture-of-Experts models accurately perceive image content yet fail in subsequent reasoning . Seeing but not thinking phenomenon is a puzzling phenomenon . |
| Approach: | They propose a routing-guided intervention method that enhances domain expert activation. |
| Outcome: | The proposed method achieves consistent improvements on visual reasoning tasks. |
ProReason: Multi-Modal Proactive Reasoning with Decoupled Eyesight and Wisdom (2025.emnlp-main)
Copied to clipboard
Jingqi Zhou, Sheng Wang, Jingwei Dong, Kai Liu, Lei Li, Jiahui Gao, Jiyue Jiang, Lingpeng Kong, Chuan Wu
| Challenge: | Large vision-language models often prioritize language knowledge over image information on visual reasoning tasks, incurring performance degradation. |
| Approach: | They propose a visual reasoning framework that decouples vision-reasoning capabilities and multi-run proactive perception. |
| Outcome: | The proposed framework outperforms existing models on benchmarks for open-source and closed-source models with 13.2% performance gain. |
Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models (2026.acl-long)
Copied to clipboard
Woody Haosheng Gan, Deqing Fu, Julian Asilis, Ollie Liu, Vatsal Sharan, Robin Jia, Willie Neiswanger
| Challenge: | Steering methods have emerged as effective tools for guiding large language models’ behavior, yet multimodal large language model lacks comparable techniques due to architectural diversity and limited availability of multimodal steering vectors. |
| Approach: | They validate steering vectors derived solely from text-only LLM backbones and use a cross-modal transfer technique to reuse existing interpretability tools. |
| Outcome: | The proposed steering vectors can guide and enhance multimodal models using SPAR, Mean Shift, and Linear Probing. |